# Multi-modal LLM

HPT (Hyper-Pretrained Transformers) is a novel multi-modal large language model framework introduced by the HyperGAI research team. It enables the efficient and scalable training of large multi-modal foundation models, capable of understanding various input modalities including text, images, and videos. The HPT framework can be trained from scratch or efficiently fine-tuned using existing pre-trained vision encoders and/or large language models.

RPG-DiffusionMaster is a novel zero-shot text-to-image generation/editing framework that leverages the chaining reasoning ability of multi-modal LLMs to enhance the composability of text-to-image diffusion models. This framework utilizes an MLLM as the global planner, decomposing the complex image generation process into multiple simple generation tasks within subregions. Simultaneously, it proposes complementary regional diffusion to achieve compositional generation. Furthermore, the proposed RPG framework integrates text-guided image generation and editing in a closed-loop manner, augmenting its generalization capability. Extensive experiments demonstrate that RPG-DiffusionMaster outperforms state-of-the-art text-to-image diffusion models such as DALL-E 3 and SDXL in multi-category object composition and text-image semantic alignment. Notably, the RPG framework exhibits broad compatibility with diverse MLLM architectures (e.g., MiniGPT-4) and diffusion backbones (e.g., ControlNet).